As a child, Grimes took up the violin, then began playing tuba, English horn, percussion, and finally the double bass at age 13 or 14, while he was in high school. Grimes furthered his musical studies at The Juilliard School, and established a reputation as a versatile bassist in the mid 1950s. He recorded or performed with saxophonists Gerry Mulligan and Sonny Rollins, pianist Thelonious Monk, singer Anita O'Day, clarinetist Benny Goodman and many others.